Skip to content

FEAT Configure scan targets before launch - #3926

Closed
Richard Lundeen (richlundeen) wants to merge 10 commits into
microsoft:mainfrom
richlundeen:richlundeen-explicit-scan-targets
Closed

Richard Lundeen (richlundeen) wants to merge 10 commits into
microsoft:mainfrom
richlundeen:richlundeen-explicit-scan-targets

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

Let users configure a scan before connecting a model. Browsing scenarios and selecting techniques no longer requires OPENAI_CHAT_MODEL or creates an OpenAI target behind the scenes.

  • Create or select the target under test, scorer, and any needed scoring/adversarial targets from the launch form. Creating or cancelling returns to the same draft.
  • Check each technique's requirements before work starts. Deterministic scorers need no scoring model; techniques without generation need no adversarial model. Missing or incompatible selections produce clear configuration errors.
  • Use explicit selections or configured defaults consistently across Python, CLI, API, and GUI. Environment-based initializers still work; one model serves multiple roles only when selected for each role.
  • Validate all suite children before launching any, and check saved component identities before resume so changed defaults cannot silently switch targets.
  • Update setup/scanner guidance and configuration/resiliency examples to explain role selection while retaining environment-based initialization.

Scanner with OPENAI_CHAT_MODEL unset

Both screenshots are real production builds with the variable unset and no configured targets or scorers. Before is the pre-change baseline (80f28af); after is this PR (99d7997). No scan or provider request was submitted.

Before After
The scanner is blocked by “Environment variable OPENAI_CHAT_MODEL is required.” The setup form opens: draft the objective and create/select targets and scorers. Launch stays disabled until required selections are supplied.
Before: scanner blocked because OPENAI_CHAT_MODEL is unset After: scanner setup opens with OPENAI_CHAT_MODEL unset and offers target and scorer creation

Decouple scenario discovery from runtime roles, validate selected dependencies before admission, and preserve explicit setup and saved identities across Python, API, CLI, GUI, and suites.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 4, 2026 09:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Explicit scoring targets can be silently ignored, standard seeded tests fail without an optional environment variable, and readiness checks miss target requirements and changes to execution dependencies.

7 open issues
What changed in this PR

This PR makes scan dependencies explicit across PyRIT’s Python, CLI, API, and GUI entry points, removing implicit OpenAI helper construction while retaining deliberately configured defaults.

Changes:

  • Resolve scoring and generation dependencies lazily, with readiness checks before execution.
  • Add suite preflight, saved dependency references, and in-place target/scorer setup.
  • Update regression coverage and migration examples for explicit role configuration.
File Description
tests/​unit/​setup/​test_technique_initializer.py Verify discovery avoids implicit provider construction.
tests/​unit/​setup/​test_load_default_datasets.py Check provider-free scenario metadata discovery.
tests/​unit/​scenario/​scenarios/​adaptive/​test_text_adaptive.py Update explicit helpers and resume expectations.
tests/​unit/​scenario/​garak/​test_latent_injection.py Update dependency-change error expectation.
tests/​unit/​scenario/​garak/​test_figstep.py Assert retained scorer configuration.
tests/​unit/​scenario/​garak/​test_audio_achilles_heel.py Assert retained scorer configuration.
tests/​unit/​scenario/​foundry/​test_red_team_agent.py Cover deferred scorer setup and explicit helpers.
tests/​unit/​scenario/​core/​test_scenario.py Account for persisted role references.
tests/​unit/​scenario/​core/​test_scenario_target_defaults.py Verify configured defaults replace provider fallbacks.
tests/​unit/​scenario/​core/​test_scenario_readiness.py Add role readiness and dependency-drift tests.
tests/​unit/​scenario/​benchmark/​test_adversarial.py Update lazy scorer expectations.
tests/​unit/​scenario/​airt/​test_scam.py Update explicit helpers and deferred configuration.
tests/​unit/​scenario/​airt/​test_psychosocial.py Verify sub-harm scorers remain lazy.
tests/​unit/​scenario/​airt/​test_leakage.py Update target identity and lazy scorer assertions.
tests/​unit/​scenario/​airt/​test_jailbreak.py Update lazy scorer identity expectation.
tests/​unit/​scenario/​airt/​test_cyber.py Update lazy scorer identity expectation.
tests/​unit/​models/​test_scenario_suite_execution.py Distinguish preview and launch target requirements.
tests/​unit/​cli/​test_pyrit_scan.py Check helper-role flag parity with the shell.
tests/​unit/​backend/​test_target_service.py Test explicit connection-field validation.
tests/​unit/​backend/​test_scenario_suite_execution_service.py Cover rejection before suite child admission.
tests/​unit/​backend/​test_scenario_service.py Update lazy helper-resolution expectations.
tests/​unit/​backend/​test_scenario_run_service.py Cover modern role references and initialization metadata.
tests/​end_to_end/​browser_initializer.py Register offline targets, presets, and suites.
pyrit/​scenario/​scenarios/​technique/​execute.py Defer default scorer resolution.
pyrit/​scenario/​scenarios/​garak/​propile.py Use the resolved scorer directly.
pyrit/​scenario/​scenarios/​garak/​prompt_inject.py Use the resolved scorer directly.
pyrit/​scenario/​scenarios/​garak/​figstep.py Build scoring configuration during attack construction.
pyrit/​scenario/​scenarios/​garak/​doctor.py Defer default scorer resolution.
pyrit/​scenario/​scenarios/​garak/​divergence.py Use the resolved scorer directly.
pyrit/​scenario/​scenarios/​garak/​audio_achilles_heel.py Defer attack scoring configuration.
pyrit/​scenario/​scenarios/​garak/​api_key.py Simplify resolved scorer access.
pyrit/​scenario/​scenarios/​foundry/​red_team_agent.py Resolve helpers according to selected compositions.
pyrit/​scenario/​scenarios/​benchmark/​adversarial.py Defer default scorer resolution.
pyrit/​scenario/​scenarios/​airt/​scam.py Defer scoring and generation configuration.
pyrit/​scenario/​scenarios/​airt/​rapid_response.py Defer default scorer resolution.
pyrit/​scenario/​scenarios/​airt/​psychosocial.py Lazily prepare and bind selected sub-harm scorers.
pyrit/​scenario/​scenarios/​airt/​multilingual.py Preserve explicit generation targets and defer scoring.
pyrit/​scenario/​scenarios/​airt/​leakage.py Defer default scorer resolution.
pyrit/​scenario/​scenarios/​airt/​jailbreak.py Defer default scorer resolution.
pyrit/​scenario/​scenarios/​airt/​cyber.py Defer default scorer resolution.
pyrit/​scenario/​scenarios/​adaptive/​adaptive_scenario.py Expose adaptive factories to readiness checks.
pyrit/​scenario/​core/​scenario.py Add lazy roles, readiness checks, and dependency references.
pyrit/​scenario/​core/​scenario_target_defaults.py Remove implicit OpenAI helper fallbacks.
pyrit/​registry/​scenario_suite_resolver.py Forward explicit scoring roles through suite resolution.
pyrit/​registry/​components/​scenario_registry.py Add pre-initialization configuration validation callback.
pyrit/​models/​scenario_suite_execution.py Separate suite preview and launch requests.
pyrit/​models/​scenario_readiness.py Define shared readiness, binding, and issue models.
pyrit/​models/​catalog/​scenario.py Add explicit scoring-role request fields.
pyrit/​models/​__init__.py Export the suite readiness request lazily.
pyrit/​cli/​pyrit_shell.py Forward explicit helper-target selections.
pyrit/​cli/​pyrit_scan.py Add scoring and adversarial target flags.
pyrit/​cli/​_cli_args.py Parse helper-target flags in shell commands.
pyrit/​backend/​services/​target_service.py Require metadata-declared connection fields.
pyrit/​backend/​services/​scorer_service.py Expose objective-scoring compatibility.
pyrit/​backend/​services/​scenario_suite_execution_service.py Preflight children and carry dependency references.
pyrit/​backend/​services/​scenario_service.py Forward scoring targets into scenario configuration.
pyrit/​backend/​services/​scenario_run_service.py Validate and persist effective launch dependencies.
pyrit/​backend/​services/​scenario_configuration_resolver.py Resolve named roles and preview readiness.
pyrit/​backend/​routes/​scenarios.py Add scenario readiness endpoint.
pyrit/​backend/​routes/​scenario_suites.py Add suite readiness endpoint.
pyrit/​backend/​models/​scorers.py Add objective-scoring support metadata.
frontend/​src/​utils/​targetIdentity.ts Label resolved target-role bindings.
frontend/​src/​types/​index.ts Add readiness and explicit scoring-role types.
frontend/​src/​services/​api.ts Add readiness and scorer-creation API calls.
frontend/​src/​components/​Scorers/​CreateScorerDialog.tsx Add metadata-driven objective scorer creation.
frontend/​src/​components/​Scorers/​CreateScorerDialog.test.tsx Cover scorer creation and failure states.
frontend/​src/​components/​ScenarioSuites/​LaunchSuiteDialog.tsx Add readiness gating and in-place role setup.
frontend/​src/​components/​ScenarioSuites/​LaunchSuiteDialog.test.tsx Test readiness-gated suite launch.
frontend/​src/​components/​ScenarioSuites/​LaunchSuiteDialog.styles.ts Prevent dialog content from shrinking.
frontend/​src/​components/​Scenarios/​ScenarioDetail.tsx Add explicit role setup and readiness-gated launch.
frontend/​src/​components/​Scenarios/​ScenarioDetail.test.tsx Cover readiness errors and stale responses.
frontend/​src/​components/​helpDefinitions.ts Clarify explicit generation-role selection.
frontend/​src/​components/​Config/​CreateTargetDialog.tsx Return created targets and honor endpoint metadata.
frontend/​src/​components/​Config/​CreateTargetDialog.test.tsx Cover optional endpoint configuration.
frontend/​src/​components/​Chat/​ChatScorerPicker.tsx Support resolved default-scorer labels.
frontend/​e2e/​explicit-targets.seeded.spec.ts Add offline scan and suite setup acceptance flows.
doc/​scanner/​0_scanner.md Document explicit helper-target CLI flags.
doc/​code/​setup/​2_resiliency.py Configure scoring explicitly in the retry example.
doc/​code/​setup/​2_resiliency.ipynb Mirror explicit scoring setup and annotate historical output.
doc/​code/​setup/​1_configuration.py Explain deliberate environment-based initialization.
doc/​code/​setup/​1_configuration.ipynb Mirror configuration migration guidance.
doc/​code/​setup/​0_setup.md Document role configuration, readiness, and migration.
doc/​code/​scenarios/​2_custom_scenario_parameters.py Update scenario import and explain role prerequisites.
doc/​code/​scenarios/​2_custom_scenario_parameters.ipynb Mirror import and explanatory updates.

Review effort: Balanced

Comment thread pyrit/scenario/core/scenario.py Resolved
Comment thread pyrit/scenario/core/scenario.py Resolved
Comment thread pyrit/scenario/core/scenario.py Resolved
Comment thread pyrit/scenario/core/scenario.py Resolved
Preserve explicit API connection-field validation in the imported HuggingFace event-loop regression by supplying its test-only token.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bind benchmark and converter dependencies, align effective factories and actual consumer capabilities, preserve scorer rubrics, and repair launch correction/confirmation UX. Cover frozen suite references at real registry admission.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retain positive reuse and deliberate fresh edit-reruns, assert earlier drift rejection before datasets or new results, and configure the helper explicitly when constructing synthetic old saved runs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Check every actual shared-helper consumer before work, including accepted simulated seeds with uses_adversarial=False. Preserve native helper success, objective-incompatible SKIP behavior and configured translation adaptation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen

Richard Lundeen (richlundeen) commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

In response to this review.

Addressed the seven causes summarized here in 962b86d: seeded output fallback, explicit scoring selection, recoverable scoring controls, factory parity, converter identities, actual helper capabilities and benchmark-owned roles. This includes the subsequently identified simulation bypass when the outer adversarial flag is explicitly false. The repair also adds the missing nonempty suite admission regression and groups repeated suite remediation. Local role-confirmation/model-help observations remain separately framed.

Confirmed at final published head 1e56308. The subsequent change is seeded-test-only: deferred provenance fixture setup and test-owned registry/explicit offline execution assertions. The full normal seeded shard passed (112 tests, evidence directory unset). Prior source fixes and their recorded visual evidence are unchanged; those earlier visual checks were not rerun.

Defer packaged compatibility reads until fixture execution. Compare complete target registry identity deltas and persisted offline run dependencies instead of prohibiting other specs' explicit OpenAI-shaped registrations.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.

@richlundeen Richard Lundeen (richlundeen) changed the title [BREAKING] FEAT Require explicit scan target roles FEAT Configure scan targets before launch Nov 4, 2026
Keep setup focused on selecting only needed scan roles and preserving environment-based initialization. Remove implementation and review-detail exposition.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve explicit scan roles through launch-form extraction and scenario run guards. Update readiness consumer mocks without changing production fallbacks or presentation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Update paired documentation, suite contracts and readiness fixtures to the public import policy without behavior or notebook output changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Assert disabled launch and actionable missing-target readiness before selecting the objective target. Preserve target-free estimates, direct API rejection and exact dataset execution checks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.

@richlundeen

Copy link
Copy Markdown
Contributor Author

Superseded by #3794, which merged with these changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants